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Abstract 

This article provides lessons learned about community building and child assessment training 
from an external evaluation of an Arizona statewide initiative for early childhood programs. 


Children’s success in school is largely affected by their experiences during the first few years of 
their lives. Research indicates that during the first three years, the young child’s brain 
experiences dramatic growth. Zero to Three, a nonprofit organization whose mission is to 
improve the lives of infants and toddlers, states that “there are many ways parents and caregivers 
can help children get off to a good start and establish healthy patterns for life-long learning 
(Behavior and Development, Brain Development, n.d.). 

To further emphasize the importance of the early years, the executive summary of The National 
Academies provided in From Neurons to Neighborhoods: The Science of Early Childhood 
Development highlights core concepts of development in young children. These concepts, in 
part, include the following (Shonkoff & Phillips, 2000, pp. 3-4): 

1. Children are active participants in their own development, reflecting the intrinsic human 
drive to explore and master one's environment. 

2. The development of children unfolds along individual pathways whose trajectories are 
characterized by continuities and discontinuities, as well as by a series of significant 
transitions. 

3. The timing of early experiences can matter, but, more often than not, the developing child 
remains vulnerable to risks and open to protective influences throughout the early years 
of life and into adulthood. 

4. The course of development can be altered in early childhood by effective interventions 
that change the balance between risk and protection, thereby shifting the odds in favor of 
more adaptive outcomes. 

First Things First 

Due to the great importance of the early years of children’s development, an initiative was 
brought before the voters of Arizona in 2006, to support a proposition to fund quality early 
childhood development and health, operating under the following principles (History, n.d.): 
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1. Local communities must come together to plan and administer what works best in 

their community. 

2. It must be flexible enough to accommodate the unique demographics of our state. 

3. It must be transparent and held accountable for outcomes. 

The Arizona voters passed the proposition in, what is referred to as, a statistical landslide. As a 
result, a state level board, known as the Arizona Early Childhood Development and Health 
Board, was created for the purpose of improving the lives of Arizona’s youngest citizens. The 
Board adopted the name First Things First (FTF), which exemplifies the importance of early 
childhood development. The mission of FTF is to increase the quality of and access to early 
childhood development and health programs that ensure a child entering school comes healthy 
and ready to learn. FTF is a community-based initiative and funds Regional Partnership Councils 
to provide services based on community needs. One principle of the proposition included that the 
initiative must be transparent and be held accountable for outcomes. In turn, this principle 
manifested the need for an external evaluation. 

External Evaluation 

In 2007, three Arizona universities formed the Tri-University consortium to begin an external 
evaluation team to evaluate the system of FTF. This external evaluation team was tasked with 
determining whether or not there is a positive increase in the educational and health outcomes for 
young children, birth to age five, and their families, as well as increased capacity in the early 
childhood service delivery system as a result of the FTF statewide initiative. That is, the team 
would evaluate whether or not programs funded by FTF were improving the general wellbeing of 
children in Arizona and their preparedness for beginning kindergarten. To accomplish this task, 
the external evaluation team developed a mixed-methods evaluation to be conducted across the 
state of Arizona. This evaluation included an accelerated longitudinal study of child outcomes 
and a cross-sectional annual evaluation of kindergarten readiness (approximately 9,000 and 
1,200 children, respectively). To assess school readiness, the evaluation team used direct child 
assessments, looking at language, literacy, mathematics, motor skills, and health. Additionally, 
questionnaires and surveys were used to collect information on socio-emotional skills, literacy, 
math skills, and overall health. 

As stated by the consortium (Tri-University Consortium, 2008), 

First Things First (FTF) provides an unprecedented opportunity to invest in the lives of 
Arizona’s young children and the evaluation of FTF provides an unprecedented 
opportunity to study how improvements in access to and quality of educational and health 
services impact young children’s development, (p. 7). 

A study of this magnitude to determine the impact of statewide programs provided ample 
opportunity for evaluators to utilize multiple approaches to build community and to assess young 
children and their readiness for kindergarten. 

The focus of this article is to discuss the lessons learned or the assembling of “the puzzle” from 
working on a large-scale study. As such, no data interpretation will be included. As the team 
from Northern Arizona University (NAU) self-reflected on the first three years of this study, we 
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gained valuable insight beyond the initial intent of the study to evaluate FTF. A primary lesson 
or first piece of the puzzle was the ability to nurture a partnership within the communities 
involved in the project, including partnerships with local school administrators, teachers, and 
kindergarteners and their families. NAU was charged with a data-collection area that consisted of 
nearly half the state. NAU has a long history of partnerships in northern Arizona and continuing 
those partnerships was a high priority for us. It is important to note that much of northern 
Arizona is classified as rural, with areas consisting of tribal land, mixed with many remote, small 
towns. The university’s location, in Flagstaff, is one of the few larger cities in northern Arizona, 
and still only has a population of about 60,000. The College of Education at this university is 
committed to meeting the educational needs of remote communities by offering programs in 
nontraditional formats through online courses and face-to-face programs offered in various 
communities. Our desire to reach outlying communities is shown through our commitment to 
ongoing partnerships. For this reason, the evaluation team based at NAU felt strongly about 
carrying out this study in a manner that showed respect and sensitivity to the communities in our 
data-collection area. 

This, in part, was accomplished through the next piece of the puzzle, which involved qualified 
community members joining the university team as data collectors. This built upon the already 
well-established relationships between the university and surrounding communities in northern 
Arizona. It was important to recruit data collectors who would be good with children, which is 
why most of the data collectors were current/former teachers or people who had worked closely 
with young children in other ways, such as school counselors or librarians. While the team was 
primarily located in the Flagstaff area, we made efforts to recruit data collectors in many of the 
more remote locations where we would be assessing children. We began by contacting educators 
or former educators we knew from our interactions with local schools and invited suggestions 
from them for people they knew in the more remote communities who might be interested in 
working with us. As word got out about our project, we received calls from educators, 
expressing a desire to join the team. Having local data collectors added credibility to our project 
and resulted in a greater trust with our partnering communities. In addition, it was our goal to 
hire some data collectors who were fluent in Spanish or in Navajo (Dine') in order to ensure that 
we could communicate effectively with all children and families participating in the study. While 
finding versions of our child assessments that were in languages other than English was not 
always possible, we were able to use Spanish language versions for some of the assessments. 

Another important piece of the puzzle was the consideration for building community and 
partnerships related to our recruitment of schools that we hoped would agree to participate in the 
kindergarten readiness study. We felt that it was important for our recruiting efforts to be done 
by people who were familiar with the culture of school. For this reason, those on the team who 
had formerly worked in the school system made initial contacts with assistant superintendents 
and principals. We found this to be extremely successful as the recruiters and school 
administrators seemed to “speak the same language,” expressing those concerns that would 
matter most to the schools about their participation. It was important to us not to treat our 
partners in a way that made them feel like we were merely using them for data collection, but 
rather that the study would allow their voices to be heard and how their participation would 
benefit Arizona schools, children, and families. 
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Finally, once we enlisted the participation of schools, families, and children, we were committed 
to taking the time necessary to help participants understand the full data-collection process. We 
wanted teachers and families to be comfortable with the project and to understand the incentives 
they would receive for their participation (e.g., big books for teachers, gift cards and books for 
families). We were also committed to scheduling our assessments in a manner that would cause 
the least amount of disruption to the kindergarten teachers. Data collectors were careful to 
communicate openly with the teachers about the amount of time they would be working with 
children and were committed to completing the assessments in a timely manner. For this reason, 
we committed to spending additional time and money to send data collectors to various 
communities ahead of time to meet with teachers and families to explain the project and answer 
their questions, rather than just showing up on the day of data collection. Moving with this 
deliberate and respectful approach meant that we were not always quite as quick as our 
consortium colleagues to begin data collection, but our target numbers were always met. 

Overall, it was important to us to take the time necessary up front to meet with school 
administrators and teachers to be sure they were supportive of our efforts. Because of this, we 
found that most of the schools and teachers were happy to participate and eager to have us return 
in the future. This was extremely important to us in that we hoped to return to schools in 
subsequent years to evaluate the readiness of kindergarteners and these strong partnerships were 
a key to the success of our future evaluation efforts. 

Teacher Education Program 

Through our work on this evaluation study over a period of three years, we also learned lessons 
about how our involvement in this study benefited the teacher candidates in our early childhood 
education programs. This project ultimately provided a key puzzle piece being a literature 
resource about the use of various child assessments, including decisions made about assessment 
selection and administration. Selection was, in part, based on the ability to engage children 
through game-like, yet psychometrically-sound assessments and the administration of 
assessments was improved upon through lessons learned during training of data collectors and 
administration of the assessments in the field. For example, we found during our pilot study that 
the math assessment originally selected was too material-intensive and time-intensive, requiring 
an amount of time that was simply too much for the attention span of young children. We 
suggested changes to the consortium and later replaced the assessment tool with one that would 
provide the data we needed, but in a manner that was more respectful of children. 

Additionally, in the team’s ongoing effort to address the goal of self-evaluation set by the 
National Council for Accreditation of Teacher Education (NCATE, 2008), the information 
gained from this project served to inform our early childhood programs about effective 
community-building practices. The National Association for the Education of Young Children 
(the leading organization for early childhood) provides standards for teacher education programs 
and Standard 2 (NAEYC, 2010) addresses the building of family and community relationships. 
Our involvement in the study strengthened our commitment to this goal as we have learned first¬ 
hand effective partnering strategies, which can be shared with future teachers and ultimately 
benefit young children. 

Recommendations 
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Longitudinal research is noted as having many strengths in measuring changes and temporal 
order of relationship in variables allowing for stronger casual interpretations than in cross- 
sectional designs (Ribsel, Walton, Mowbray, Luke, Davidison, & Bootsmiller, 1996). The rarity 
of longitudinal studies is in part the challenge of participant recruitment and participant retention. 
We experienced both of these challenges in our study. From our experiences and lessons learned 
on this large-scale longitudinal study, we offer the following strategies for both recruitment and 
retention of participants for longitudinal studies. 

1. Trust Building: At the forefront of recruitment and the key to a successful longitudinal 
study is the development of trusting relationships between the researchers and the 
participants (McGregor, Parker, LeBlanc, & King, 2010). You must know your target and 
accessible audience. Here, we had the advantage of team member’s experiences in school 
settings. Rodriguez, Rodriguez, and Davis (2006) remind us that it is less difficult to 
recruit European American or White participants as compared to ethnic minorities. Our 
recruitment strategies took into account the diverse populations of northern Arizona. 

2. Snowballing for Increased Participation: Another strategy that we employed was 
snowballing —this word of mouth approach assisted us in getting our request out across 
the large area in which we were charged to study. 

3. Pilot Study: We strongly suggest that a pilot study be perfonned. We used this strategy to 
review protocols and timing. We also had regular team meetings to review our 
recruitment strategies and were ready to make changes as necessary, for example, noting 
that we were limited in the number of a particular demographic—we sought out schools 
where we could gain those voices. 

4. Retention Strategies: Not surprising, at the top of the retention strategies is also trust 
along with clearly explaining the purpose of a study. “Participants are more likely to 
remain in a study if they understand the importance and relevance of the study” (Davis, 
Broome, & Cox, 2002, p. 48). Respect of one’s time also plays a key role in retention. As 
we learned from our piloting, we were able to more accurately convey to our participants 
the amount and type of commitment they would make. Part of our retention success was 
that we had a single contact person for our participants to interact with. We created an 
extensive tracking system with participant information to allow us to better interact with 
the participants. As we were working with schools, children, and families, we had to be 
flexible with our scheduling and make our visits as convenient as possible. We provided 
reasonable incentives for participants. And above all we continued to show our 
appreciation for the individual’s participation through respecting their time and cultural 
differences. 

Assembling the Puzzle 

As we look to the years ahead and our continued work within our neighboring communities, we 
see our efforts as a sort of puzzle, with each piece working and interlocking with the next. Like a 
puzzle, we quickly learned that every piece is needed to make the puzzle complete. Our work 
necessitated the cooperation of communities, schools, administrators, teachers, families, and 
children. We also learned the importance of having a great evaluation team, from the managing 
component to the very crucial field team. Without the willing participation of all, we would not 
have been able to inform First Things First about the best ways to help children in our state. In 
part, based on the foundational efforts of the university, this evaluation team found it extremely 
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valuable and surprising that even with such a large geographic data-collection area, we were able 
to work within, maintain, and expand a sense of small community-like partnerships. We feel 
fortunate to work within these communities, which allowed us to provide their voice as part of 
this external evaluation. 

Puzzles come in all shapes and sizes, as well as levels of difficulty. One might see a large-scale 
longitudinal study as the highest level of difficulty, especially if it includes so many pieces, e.g., 
children, parents, teachers, and community leaders. We approached our puzzle one piece at a 
time while never losing sight of what the final puzzle should/could look like. While there are 
several strategies for a successful study, one should always have the big picture in mind. In the 
forefront of our recommendations is the building of trust with the communities with whom you 
will be working. From that trust, came our successes. 
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